Correlating global trends in COVID-19 cases with online symptom checker self-assessments

Background Online symptom checkers are digital health solutions that provide a differential diagnosis based on a user’s symptoms. During the coronavirus disease 2019 (COVID-19) pandemic, symptom checkers have become increasingly important due to physical distance constraints and reduced access to in-person medical consultations. Furthermore, various symptom checkers specialised in the assessment of COVID-19 infection have been produced. Objectives Assess the correlation between COVID-19 risk assessments from an online symptom checker and current trends in COVID-19 infections. Analyse whether those correlations are reflective of various country-wise quality of life measures. Lastly, determine whether the trends found in symptom checker assessments predict or lag relative to those of the COVID-19 infections. Materials and methods In this study, we compile the outcomes of COVID-19 risk assessments provided by the symptom checker Symptoma (www.symptoma.com) in 18 countries with suitably large user bases. We analyse this dataset’s spatial and temporal features compared to the number of newly confirmed COVID-19 cases published by the respective countries. Results We find an average correlation of 0.342 between the number of Symptoma users assessed to have a high risk of a COVID-19 infection and the official COVID-19 infection numbers. Further, we show a significant relationship between that correlation and the self-reported health of a country. Lastly, we find that the symptom checker is, on average, ahead (median +3 days) of the official infection numbers for most countries. Conclusion We show that online symptom checkers can capture the national-level trends in coronavirus infections. As such, they provide a valuable and unique information source in policymaking against pandemics, unrestricted by conventional resources.


It remains unclear why the authors team has chosen Germany and US to include in
.
Within the text, we stated that these countries are merely examples, one for which there is a high correlation (Germany) and one for which there is a low correlation (USA). We felt that an example of the two data sources overlapped provided understanding of the work. However, we wished not to be accused of cherry-picking by featuring only a good example; thus, we displayed the two countries mentioned above.
However, and likely the source of the reviewer's comment, the explanation regarding this selection, and the associated discussion, was separate from the first introduction of Figure 1. We suggest that this separation led to the reviewer's confusion. The relevant text has been shifted such that we give the context for the country selection alongside the Figure's first introduction.

Also it would be good to not only introduce the mean/median correlation of all included countries but instead add it for each country (e.g. in a table)
We have added the requested table to the supplementary information and included reference to it within the manuscript.

Results + Discussion:
This section mixes up results and discussion. You need to strictly divide between the results based on your methods and then discuss them. Currently, you mix results and discussion. Also, the sequence is still confusing here. Still needs rework here.
We apologise for the lack of clarity here in our previous manuscript. In light of this comment, we implemented a significant restructuring of the manuscript to make a clear distinction between our results and their discussion. This restructuring includes introducing a dedicated discussion section whereby we comment upon the results and what may be driving said observations.

Confidence interval is missing for the regression line.
We have now added confidence intervals for all figures containing regressions ( Figures 1C-1F). Please note that we had already reported significance values (p-values) for these models within the text to indicate the goodness of fit.
Correlation shift results section needs further improvement. The correlation improvement by country is very small and without the absolute value of the correlation for each country without the shift not useful.
As requested above, we have added a table containing the correlation for each country as a table within the supplementary information. Further, this table also includes the maximum correlation obtained when we apply a shift and the shift with which this occurs. We reference this table within the text, and the information is used throughout the discussion.
Conspicuous is the largest increase of 0.103 for Spain, that you unfortunately did not take into account for any discussion.
We thank the reviewer for this observation. After investigation, we added the following discussion to the manuscript: "While the correlations increased universally with the introduction of a shift, for some countries, the optimal shift was negative. For example, for Spain, which exhibited the largest increase in correlation (+0.103), the optimal shift was -12 days, i.e., the COVID-19 pandemic progression precedes the trends observed in the chatbot. We believe this is due to the number of recorded cases in Spain being almost monotonic over the period analyzed. Other countries experience an initial peak followed by a Summer season with fewer infections. For Spain, infection numbers were instead stable after the initial onset. The absence of an initial peak was likely due to limited testing capacity when Spain was one of the epicentres of the pandemic's first wave in Europe [27]."

Figures: still need rework, Figure 1 (A) + (B): x-axis labels missing,
We have added labels to Figure 1B. As Figure 1A shares an x-axis with Figure 1B, we have chosen to drop the axis labels as the elements do not add new information to the graphic.

Figure 1 (C) -(F): x-axis labels + scale is not explained before in detail,
As suggested by the reviewer above, we added further explanation of these metrics to our Materials & Methods section, including a complete description of the respective scales. To aid the reader, we have also briefly reiterated what each metric represents, when the name is unclear, within the Results & Discussion.
you also decided to cut x-axis e.g. GINI score was introduced from 0 -1 but not reflected in the Figure. Plotting the complete range of the GINI score and other QoL metrics is unreasonable , considering that the subset of countries featured in our study appears only in a small region. Including everything would lead to an uninformative visualisation; the data is forced together, even overlapping. However, we understand that limiting the axis to that of the subset could be used to present a false narrative. We prevent this situation by supplying the regression for each metric against the correlation, including the significance, which objectively indicates the strength of relationships to the reader.

You mention the 4 correlations 0 or smaller in the text for GINI but not this other plots (D), (E), (F)
We are unsure about the reviewer's comment as the previous iteration of our manuscript did not contain a discussion about the subset of correlations that were negative and how they related to any of the QoL metrics, including the Gini index. The negative correlations and potential reasons why the respective countries diverged from the chatbot trends were discussed in detail but independently from the QoL metric analysis. Regardless, we hope that the extensive revision of the results and discussion section, as mentioned above, has provided additional clarity to this section and resolved the review's concern.
In addition to the changes based on the reviewer's comments, we want to draw attention to a slight correction we made to the manuscript regarding our data pipeline. We stated previously that we filtered countries which lacked data for even a si ngle day. This was incorrect. Instead, we filtered countries for which less than 200 of the 208 days had data, i.e., more than eight days missing. For reference, eight days is less than 4% of the analysed time period. Zeros were inserted for days without values. Please note that this had no downstream effect, as the previous manuscript versions were already based on this filter. Further, during this review, the manuscript underwent another editing process which resulted in a couple of small changes. The last notable change is that the author list has been reordered to reflect changes within our company over the revision period. We hope this change can be propagated in your system and is not a point of concern.
Lastly, we would again like to thank the reviewers for their comments. As outlined above, we have greatly improved the manuscript based on their feedback.